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ABSTRACT 



A primary concern in the development of large-scale, real-time, complex, computer- 
intensive systems is ensuring that the system meets the specified requirements. Further, the 
requirements themselves evolve and undergo many changes during the development process. In 
such a context, it is essential to maintain traceability of requirements to various outputs to ensure 
that the system meets the current set of requirements. 

An empirical study, utilizing focus group and protocol analysis techniques, was conducted 
with students from the Naval Postgraduate School in a simulated systems development 
environment. Based on the analysis of data, this study highlights several major issues that need 
to be considered in the development of a model of traceability. An initial model of traceability 
as well as recommendations for future research to refine and validate the model are presented. 
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I. INTRODUCTION 



A. GENERAL INFORMATION 

A primary concern in the development of complex, large-scale, real-time, computer- 
intensive systems is ensuring that the design of the system meets the specified 
requirements. As part of the systems development and maintenance process, many 
decisions and tradeoffs are made that affect a variety of the components. Further, the 
requirements themselves also undergo many changes and evolve during the development 
process. In such a context, it is essential to maintain the traceability of requirements to 
various outputs produced during the system’s design process, ensuring that the system 
meets the cunent set of requirements. 

The term traceability, as used in this research, refers to a technique used to provide 
relationships between requirements, design and implementation of a system (Edwards, 
1991). A comprehensive scheme for maintaining traceability, especially for complex, 
real-time systems, requires that all system components (not just software), created at 
various stages of the development process, be linked to the requirements. These 
components include hardware, software, humanware, manuals, policies, and procedures. 
In order to achieve this objective, it is essential that traceability be maintained through 
all phases of the systems development process, from the requiiements as stated, or 
contracted, by the customer, through analysis, design, implementation, and testing to the 
final product. 



Mainlaining consistency between the lequirenients and the design is especially 
critical in situations where an organization relies upon outside contractors for developing 
systems. Having a systematic way of validating that each requirement is met by the 
design is important, not only to ensure that the system performs correctly, but also to 
deteiTTiine whether contractual obligations have been met. 

The need to provide traceability is recognized in most ciiiical standards governing 
the development of systems for the U.S. Government. However, a clear definition of the 
types of information or lelationships between various system components that are part of 
a traceability scheme is lacking. For instance, the DoD-STD-2l67A specifies that 

the contractor shall document the traceability of the requirements allocated from 
the system specification to each Computer Software Configuration Item (CSCI), its 
Computer Software Units (CSUs) and from the CSU level to the Software 
Requirements Specification (SRSs) and Interface Requirements Specifications 
(Walters, 19yi),(DoD, 1988). 

An elaboration of this requirement states that 

the Software Design Document de.scribes the allocation of requirements from a 
CSCI to its CSCs and CSUs (Walters, 1991) 

It should be noted that even this elaboration is not specific about the nature of traceability 

linkages to be maintained. Neither the standards that lequire traceability as a part of any 

systems development effort nor the current literature elaborate on the specific types of 

traceability linkages to be maintained. Though current tools provide mechanisms to 

represent various types of linkages between system components, the interpretation of the 

meanings of such linkages is left to the user. Unless the semantics of the traceability 

linkages are clearly specified, the existence of a link between a design element and a 
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requirement could denote one of several possibilities; the requirements have been 
completely allocated, some of the design elements partially satisfy a requirement, the fact 
that the design element satisfies a requirement can be formally verified etc. 

Finally, the focus of much of existing research is on providing traceability at the 
level of software design, rather than at the level of system design. 

B. OBJECTIVE OF THE RESEARCH 

The goal of our research program is to develop a model of traceability at the level 
of systems design, relating requirements to all system components. Such a model should 
provide the semantics of the various traceability linkages or relationships between 
requirements and various system components. It should also provide mechanisms for 
reasoning with traceability infoimation to support systems development and maintenance 
activities. 

A first step towards the development of a comprehensive model is to understand the 
critical issues that relate to the capture and use of traceability information in systems 
development. A basic premise in the current research, whose results are reported in this 
document, is that development of a model of traceability could be geared toward the 
needs of various stakeholders at different stages of the systems development process. 

A variety of stakeholders are involved in the systems development process, 
including project sponsors, project managers, analysts, designers, maintainers, testing 
personnel, and end users. The approach used in this research to identify their needs has 
been an empirical one. We have conducted a study to explore the traceability needs of 
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various stakeholders and to ideiuil'y the critical issues that need to be addressed in the 
development of a comprehensive model of traceability. This study was conducted with 
graduate students of systems analysis and design in a simulated systems development 
environment. The results of this study are being used in designing a comprehensive study 
involving real stakeholders in large scale, complex, real-time systems development efforts. 



Another objective of the current research is to evaluate different research tools for 
data collection and analysis to aid in the design of the comprehensive study. Two 
powerful research tools were employed in this research: protocol analysis to study 
problem solving behavior and focus group interviews for idea generation. 

Given the above objectives, besides the development of an initial model of 
traceability, the following questions are addre.s.sed in this research: 



• What are the critical issues that need to be addressed in the development of a model 
of traceability to support various stakeholders in systems development? 

• What are appropriate methodologies that could be used in a comprehensive study 
on traceability in complex, large scale, real-time systems development 
environments? 



C. SCOPE AND LIMITAl IONS OF THE STUDY 

The current study has employed novice systems designers in a simulated design 
setting. It should be noted that this research is designed to provide only insights into 
issues that need to be investigated further, rather than to provide conclusive results. 
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Another constraint was the lack of resources to comprehensively evaluate the current tools 
that support representing traceability information. 

D. ORGANIZATION OF DOCUMENT 

The document is organized as follows: 

Chapter II provides background infoimation on the general topic of traceability, a 
discussion of some of the curreni traceability tools available, and the uses of traceability 
in the DoD and U.S. Navy. 

Chapter III describes focus groups and protocol analysis and their applications in 
this research. Each of the techniques, and why they were used in our research, is 
explained. 

Chapter IV provides the analysis of the data collected, utilizing focus groups and 
protocol analysis techniques. It discusses the major findings and relates them to current 
literature. 

Chapter V discusses design rationale as an example of traceability highlighting some 
of the major issues discussed in the previous chapter. 

The final chapter provides an initial model of traceahility and draws conclusions 
based on research data, and makes specific recommendations resulting from the research 
effort. This chapter concludes with recommended areas for additional research. 
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II. BACKGROUND 



A. WHAT IS TRACEABILITY? 

A number of different definitions can be provided for traceability, depending on the 

context in which the temi is used. Norman Schneidewind depicts traceability as a means 

for maintenance, focusing on the maintenance phase to discover sources of eiror. He 

defines traceability as "the ability to identify the technical information which pertains to 

a systems error which has been detected during the maintenance phase and thereby trace 

the eiTor to the applicable design specifications and user requiiements" (Schneidewind, 

1982). Wheieas Schneidewind’s concern for traceability is at the software level, 

Greenspan and McGowan are concerned with the use of traceability to effect changes in 

the entire system at various levels. They offer a broad definition of traceability as being; 

a property of a system description technii)ue that allows changes in one of the three 
system descriptions’-requirements, design specillcations, implementation-to be 
traced to the corresponding portions of the other descriptions. The conespondence 
should be maintained throughout the lifetime of the system (Greenspan and 
McGowan, 1978). 



To achieve the abovementioned correspondence, Agusa, et al, postulate that two- 
way traceability is required. They label traceability as bi-directional by saying, "A 
requirements description is traceable if each portion of the description can be traced to 
an originating requirement in its predecessor, and to a successor description" (Agusa et 
al, 1984). 
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While all of ihe above dellnilions focus on change/mainlenance, other aspects of 
traceability are not emphasized. Michael Edwards offers a more generic and inclusive 
definition of traceability as a technicjue used to "provide a relationship between the 
requirements, the design, and the final implementation of the system" (Edwards, 1991). 
This definition of traceability has been used in our current iiesearch. 

B. TRACEABILITY TOOLS AND CURRENT EXPECTATIONS 

The initial concern with traceability was that of providing document traceability. 
According to Horowitz and Williamson, "Document traceability defines the existence of 
relationships between two document components" (Horowitz and Williamson, 1986). 
Traceability within documents assures that the source of information is distinguishable. 

There aie a number of existing traceability tools developed by industry. Some 
salient characteristics of the major ones, including Automated Requirements Traceability 
System (ARTS) from Lockheed. Teamwork/RQT from Cadre Technologies Inc., 
Requirements Tracer (RT) from Teledyne Brown, and Requirements and Traceability 
Management (RTM) from GEM-Marconi Ltd. are discussed below. 

One of the earliest sy.stems to capture and u.se traceability data was ARTS, a 
bookkeeping program developed to manage the requirements of a large, enor-prone 
aerospace system. ARTS operates on a data base, including systems requirements and 
their characteristics. It allows for automated tracking of mquirements as they are 
partitioned and apportioned to lower level requirements. ARTS provides upward and 
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downward traceability and data base management and output operations on requirement- 
related attributes selected by the user. Like ARTS, other current tools often focus on the 
database management issues related to maintaining links between requirements and 
differing components of the system. Tlie following are the major characteristics found in 
current traceability tools: 

• Allocation of requirements to targels/design elements 

• Parsing and grouping of functional requirements 

• Traceability between documents 

• Interface with CASE tools (e.g., Teamwork, Software Through Pictures) 

• Capture functional hierarchies 

• Keyword searches 

• Assign attributes for iiequirements or traceability relationships 

• Customized report generation 

• Graphic Interfaces 

The tools differ in the degme of suppoil provided and offer only a subset of the 
above functionalities. Current traceability tools tend merely to provide mechanisms to 
represent relationships without providing a comprehensive model of traceability to aid the 
use in defining these relationships. Also, as they lack sophisticated mechanisms to reason 
with the traceability information captured, their usefulness is severely restricted. 
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C. TRACEABILITY IN THE DOI) AND NAVY 



As one of the world’s major buyers of large-scale, computer-based systems, the 
DoD takes a surprisingly detailed approach to the dilemma of detailing systems 
requirements. DoD standards may even provide an instructive checklist for content. 

In February 1988, the Defense Department specified its requirements for systems 
development in its Military Standard DoD-STD-2167A. This standard formalizes the 
tracing of requirements (in documents) from the original set entailed by the customer, to 
the contractor’s written requirements specifications, and to the design, test procedures, and 
results. DoD-STD-2l67A mandates that requirements be traceable through the entire 
system. However, the standaril states only that traceability is required, not what 
infomiation is to be maintained to achieve this. 

The DoD currently delineates its reciuirements to contractors in documents that are 
developed by numerous specialists in a format that may be thousands of pages long. 
Having a precise method for ensuring that requirements are met hy the design is vital. 
With declining defense dollars, systems must last longer, with potential for major changes 
during their lifecycle. Therefore, a key element included in a request for proposal must 
be traceability, guaranteeing that cunent set of requirements are met by the evolving 
system. 

One of the foremost issues in developing an efficient and effective system involves 
the maintenance of consistency between requirements and design. Such consistency 
entails meeting the initial requirements and maintaining requirements, design, and 
implementation consistent throughout the entire system life-cycle. 
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The current mcihod used by die Navy to specify requiiements uses mostly a 
narrative, natural-language format with supporting diagrams and charts. Ambiguities are 
frequent as natural-language specifications are inexact. If specifications are formally stated 
and can be Uansfoinied into designs in a formal manner, traceability between 
requirements and designs is a by-product of the design process itself. However, most 
specifications are natural-language and therefore mechanisms are needed to capture 
traceability information explicitly. 

In light of some I’ecent systems malfunctions that produced catastrophic 
consequences (major telephone service shutdowns, for example), it is now commonly 
understood that changes to intricate systems can result in unforeseeable and disastrous 
effects to important national defense systems. These problems possibly could be avoided 
if collect traceability methods are used along with proper maintenance of systems. 

D. SUMMARY 

Acquiring a greater understanding of the concepts of traceability is essential. A 
major challenge in this research is the development of a model that represents and 
provides the semantics of various traceability linkages or relationships between 
requirements and systems components. 



II 



III. DATA COLLECTION METHODOLOGIES 



A. INTRODUCTION 

In order lo belier invesiigate the traceability relationships, we used a two- 
pronged approach to data collection: focus group interviews for idea generation and 
evaluation, and protocol analysis of problem solving behavior. 

This chapter discusses these two techniques and the design of the study that employed 
these two methodologies. Details of the research setting, subjects as well as the reasons 
for the use of data collection technic|ues are provided. 

B. FOCUS GROUPS 

I. What Are Focus Groups? 

Focus group interviewing is possibly the most consistent qualitative marketing 
technique used today. Marketers and the media have had much to say recently about 
focus groups. Many advertising and research agencies believe focus groups to be among 
the most valid of exploratory research tools for their purposes. Today, in many marketing 
research organizations, group interviews are nearly as common as the traditional survey 
questionnaire. 

A focus group interview is a semi-structured exchange with a small group of 
people. It is not a rigidly constructed question-and-answer session, but neither is it a free 
dialogue among group members; the group has a clear agenda. In his book. Focus 
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Groups: A Practical Guide for Applied Research . Richard Kieuger slates, "A focus group 
can be defined as a carefully planned discussion designed lo obtain perceptions on a 
defined area of interest in a permissive, non-threatening environment. The discussion is 
relaxed, comfortable, and often enjoyable for participants as they share their ideas and 
perceptions. Group members inlluence each other by responding to ideas and comments 
in the discussion” (Krueger, 1988). 

Focus groups weie originally called focused interviews. They were first used 
in the 1930s by social scientists as an alternative to the technique of using an interviewer 
with closed-ended questions and one respondent; the idea was that multiple respondents 
could make comments on issues they believed to be important while interacting with one 
another. In the 1940s, focus groups were used in the evaluation of audience responses 
lo radio programs, and the observation of the effectiveness of wartime propaganda efforts. 
In the 1990s, although much of what we know about the focus group technique has come 
from market research, all professions, from academia to diplomacy and politics, to the 
social science and business worlds, are adopting this eminently versatile method. 

The focus group interview is a highly flexible tool and as such is extremely 
popular. Focus groups arc appropriate for exploratory analysis when little is known about 
a topic; for generating ideas and research hypotheses; for determining how groups of 
individuals think about current issues; for producing information, uncovering potential 
problems, and encouraging creativity. Today, focus group interviewing is considered to 
be a valid scientific method. 
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The focus group technique was used by the Reagan administration in 1988 (an 
election year) to determine the character and extent of the knowledge/opinion gap 
between the American public and government officials, in regard to American-Soviet 
relations. The Reagan team a.skcd two suburban Philadelphia focus groups of "average 
citizens" to examine the ways in which a future Soviet-American summit meeting could 
be believably presented to the American people while simultaneously gamering popular 
support. Based on focus group responses, the team chose for the trip the theme, "A 
brighter future and a safer world for all people." Tlie Philadelphia groups also helped 
determine some of the events of the trip, and with whom President Reagan would meet. 

Focus group interviewing today usually involves seven to 12 individuals who 
discuss a particular topic under the direction of a moderator, who promotes discussion and 
ensures that the group stays on the subject. Smaller groups may be dominated by one or 
two members, while larger groups are difficult to manage, and limit participation by all 
members. A typical session will last from one and one half to two and one half hours. 

2. Uses, Pros, and Cons of Focus Groups 

Focus groups may be used as a method for testing hypotheses, especially 
when the researcher has strong reasons to believe his hypotheses are coirect. The focus 
group technique is not without its critics, who maintain that focus groups don’t provide 
"hard" data and that group members may be atypical of a larger population. But even the 
critics acknowledge that focus groups are useful for exploratory research where little is 
known about a topic. 
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The more commonly lauded uses of focus groups include: 



• Generating research hypotheses that can be submitted to fuilher research and 
testing, using more qiianiiiative approaches; 

• stimulating new ideas and cieative concepts; 

• diagnosing the potential for problems with a new program, service, or product; 

• generating impressions of products, programs, services, institutions, or other objects 
of interest; and 

• learning how respondents talk about the phenomenon of interest. 

Some advantages of focus groups include: 

• They are quicker and le.ss costly than individual interviews. 

• Direct contact with respondents allows for probing and clarification; the respondent 
can use his own words to express himself. 

• Through group interaction, members tend to inlluence and change each others’ 
opinions, and this shift can be studied; information and insights are provided that 
would not be available without the group’s interaction. 

• Focus Groups have a dynamic effect, encouraging creativity. 

• Results are believable and easy to understand. 

• There is much research and theory related to focus groups. 

Some disadvantages of focus groups include: 

• The sample size is limited. 

• Groups may vary widely in their enthusiasm levels and responses. 

• Responses are not independent and may be biased by one or more participants. 
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• Summarization and inieipieiation of responses may be difficult. 

• The moderator has less control in a group setting than in a one-on-one intei'view. 

• The moderator may bias results. 

In their hook, Foctis Groups: Theory and Practice . David Stewart and Prem 
Shamdasani state, "We should not overlook the cases in which focus groups alone may 
be a sufficient basis for decision making. One example in an applied research setting 
would be the identification of Haws or serious problems with a new product or program 
that would necessitate redesign" (Stewart and Shamdasani, 1990). 

When little is known about a particular subject, there are few good alternatives 
to focus groups. Focus groups are quicker ;md less expensive than individual interviews; 
one must simply recognize the potential for obscuring individual responses. 

3. Group Dynamics 

It is the characteristics of group members in relation to one another, and not 
just individual differentiation, that determine group behavior and performance. Focus 
groups should be structured to facilitate the goals of the researcher, while avoiding 
manipulation of the final results. 

A recurrent supposition regarding focus groups is that superior data are 
obtained when members are strangers. However, Stewait and Shamdasani state: 

Generally, focus group sessions are preceded by ’get-acquainted’ and ’warm-up’ 
sessions that usually provide participants ample opportunity to get to know one 
another. Thus, the issue of acHpiaintanceship appears to be a matter of degree in 
most focus groups, and its inlluence appears modest at best (Stewart and 
Shamdasani, 1990). 
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Another concern regarding focus groups is the members’ backgrounds. 

In general, inleiaclion is easier when individuals with similar socioeconomic 
backgrounds comprise the group. Similarity of abilities, level of intelligence, and 
knowledge lends to facilitate communication at the same wavelength. Similarly, in 
culturally and racially homogenous group situations, it may be easier to encourage 
member participation. This suggests that focus groups should be designated to 
maximize interaction by assunng similarity with respect to socioeconomic status 
(ibid, 1990). 



A highly homogenous group may be able to move through many questions 
quickly, while a highly heterogenous group may belabor even a couple of questions. But 
a small degree of variation within group characteristics is often a helpful way to obtain 
the contrast and variation that spark lively discussions. 

Krueger advises; 

The focus group technique works well when all participants are on an equal 
basis. Participants should be grouped with care. Participants should be placed with 
others at the same level or status in the organization. (Krueger, 1988). 

4, The Moderator 

When a moderator/interviewer has little experience or prior knowledge in a 
field, the focus group technirpie can be ideal. In his treatise. Focus Groups: A 

Qualitative Research. David L. Morgan states: 

When the researcher is relatively new to an area, or puts a priority on not 
repeating the received wisdom in a I'leld, focus groups have much to offer. The fact 
that group interviews can produce u.seful data with relatively little direct input from 
the researcher may be a distinct advantage, especially in comparison to other 
interviewing techniques (Morgan, 1988). 

A designated moderator/interviewer does away with much of the distraction 
associated with the group having to develop its own leadership. With respect to the 
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discussion, ihe moderator may be highly directive or very non-directive--letting the 
discussion flow naturally as long as it remains on the topic. It is quite common for an 
interviewer to start with some geneial questions, then focus on more specific issues as the 
group proceeds. 

The amount of direction provided by the interviewer does influence the type 
and quality of the data obtained from the group. The amount of stiucture and direction 
by the moderator must be determined by the broader research agenda, including types of 
infoiTuation sought, degree of detail the information requires, and the manner in which 
the infonnation will be used. 

Discussion of issues relevant to the needs of the researcher occur most readily 
when the moderator takes a more directive and structured approach. When this occurs, 
however, participants discuss what is important to the researcher, not necessarily what 
they consider significant. 

5. The Interview Guide 

In setting the agenda for a focus group, the moderator must choose from 
among research questions to create the interview guide. An alternative, available to a 
researcher conducting several focus groups, is the rolliim interview guide. The interview 
guide developed for Group One is revised and used for Group Two. Based on Group 
Two results, the guide is revised for Group Three, and so on. This technique makes the 
best use of multiple focus groups, permitting information to be refined over time as more 
information is obtained about a subject. 
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6. Analyzing rocns Croup Data 



According lo Siewiiil and Shamdasani; 

The most common purpose of a focus group interview is for an in-depth 
exploration of a topic about which little is known. For such exploratory research 
a simple descriptive narrative is quite appropriate. More detailed analyses simply 
are not necessary or efficient (Stewart and Shamdasani, 1990). 

For analyzing the content of focus groups, the cut-and-pasle method is 
immediate and cost-effective. Cut-and-paste is a useful technique, but often relies on the 
judgment of a single analyst. Usually it is pieferable to have two or more analysts code 
the focus group I'esults independently. 

C PROTOCOL ANALYSIS 

1. Delliution of Protocol Analysis 

Vitalari and Dickson define protocol analysis as "the process of translating the 
chaotic collection of infoimation, which is derived from the protocol, into more useful 
and meaningful representation" (Vitalari and Dickson, 1983). In a more general sense, 
protocol analysis can be thought of as the collection and analysis of verbal reports (called 
protocols) made by subjects while they perform a specific task. In most cases, protocol 
analysis is used to generate a mechanism for tracing a subject’s thought process. 

Ericsson and Simon distinguish between two different types of verbalization 
procedures— retrospective verbalization and concurrent verbalization. Retrospective 
verbalization refers to the technique in which the researcher asks the subject for 
infoimation about his/her thought proces.ses after the task is completed. Concurrent 
verbalization, used in this research, refers to a technique in which the subject is asked 
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simply to verbalize liis/lier thoughl process while working on a task (Eiicsson and Simon, 
1980). 

Concunent verbalization procedures have been used extensively in the study 
of human problem-solving, including such areas as general problem-solving behavior, 
physics problem-solving behavior, stock selection, pediatnc cardiology, and accounting 
information decisions (Vitalari, 1981). 

2. Validity of Concurrent Verbalization 

According to Vitalari, despite the extended use of concunent verbalization, 
considerable contention surrounds its u.se. Some researchers have questioned the validity 
of verbal reports. The lour major issues under contention are: 

• skewed verbalization of true thought process 

• incompleteness 

• interference with thought process 

• subjective bias during analysis 

The first major issue is that the subject must articulate his/her own thought 
process, he/she is allowed to decide how it will be verbalized. Therefore, the thought 
process is different than the one verhalized. The second issue, incompleteness, argues 
that the task of verbalizing interferes with the main task and hence, the subject is only 
able to verbalize a small part of the actual thought process. The third issue, interference 
with thought process, refers to the researcher probing the subject to explain his/her 
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reasoning, etc., during the expciimenl. The fourth issue, subjective bias, occurs if the 
researcher’s analysis of the data is different from what is implied by the verbalizations. 

Some of tlie ways to safeguard against the above problems include ensuring 
the researchers do not probe the subjects during the experiment, and having independent 
researchers analyze the protocols (Vitalari, 1981). 

3. Evaluating Protocol Analysis Data 

A wide range of methods to evaluate protocol analysis data is reported in 
literature, varying from a quick count of the occurrence of certain words in the protocols, 
to an extensive analysis of all the elements in the tasks under investigation. The method 
chosen to analyze the conciineni verbalizations in this research was the simple technique 
of searching through the protocols for unique ideas, thoughts, etc., relating to traceability 
issues. 



D. STUDY DESIGN 
1. Subjects 

The subjects in our study came from a Masters program in Infonnation 
Technology at the Naval Postgraduate vSchool. The study was conducted after the 
students had completed the analysis, design, and implementation of an information 
system, based on a case study conducted in a graduate level systems analysis and design 
course. The case study development was ba.sed on a real-life, large-scale project and had 
been successfully used in similar studies (Ramesh and Dhar, 1992). The case analysis 
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involved a variety ol'data-gatliei ing methods during the analysis phase, including informal 
descriptions of user needs, simulated client meetings, and actual documents from real-life 
situations. The major outputs developed by the participants included requirements 
statements, data flow diagrams, entity-relationship diagrams, database design, and 
implementation. 

2. Case Study 

The case used in this research was in the domain of customer order processing 
in a utility company. The prohlem was selected for several reasons: 



• The case study had been developed after an extensive domain analysis was 
conducted, based on a real-life system developed by a large infonnation systems 
consulting organization. 

• The case study had been u.sed succe.ssfully in several sellings, including protocol 
analysis of group prohlem-.solving beltavior. 

♦ The problem domain v/as familiar to the students, as they had personal experiences 
with the services provided by the system. 

♦ Real-life data could be easily collected from a utility company and used in the 
analysis and design of the system when necessary (e.g., rate schedules were 
collected from the local utility company and used in systems design). 

♦ The problem was sufllcienlly complex to covei all the basic elements of systems 
design. 

• The problem could be partitioned so that different groups of students could be 
assigned projects that could be completed within a reasonable time frame. 



These activities were completed during a period of over two months prior to 
the subjects’ participation in the focus groups. Many subjects had extensive experience 
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in domains oilier than compuler-ha.sed systems developmeni, such as shipbuilding and 
aviation maintenance, where concepts ol liaceahilily are widely recognized. 

Appendix A provides details of the case study including the various outputs 
produced by the subjects during the exercise. 

3. Focus Groups in our Research 

Six focus groups were conducted over a two-week period following the 
subjects’ completion of their case studies. Each group consisted of approximately eight 
to 10 subjects and each group lasted roughly one-and-one-half hours. The focus groups 
were conducted in a semiformal selting-a meeliiig room equipped with facilities for 
audio/video recording. The following steps were utilized for each session: 



• A short warm-up period, during which everyone, including the moderators, was 
introduced and the ground rules of the interview stated. 

• A predisposition discussion about the traceability issues that needed to be explored, 
including general discussions on the various stakeholders’ interest in traceability. 

• A collective and comparative discu.ssion of all topics, followed by a wrap-up of the 
discussion. During this segment, the participants were prompted for their 
summaries of what was discussed in the group meeting. 



In light of the information detailed above, we felt ourselves to be on firm 
empirical ground in using focus groups for our research. The following are specific 
reasons we used this technique: 



• The focus group is a valid, proven research tool in areas such as traceability, where 
not much is known about the topic and where generation of ideas and hypotheses 
for further study is desirable. 
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• There has been ample research on ihe focus group technique to give us a solid 
background in using it; at the same time, focus groups may be conducted infoimally 
to work well within our academic setting. 

• As mentioned above, when moderators are new to a research topic, they are actually 
at an advantage in not reiterating the established knowledge of the field. 

• The groups of students attending the Naval Postgraduate School were acquaintances 
who have similar socioeconomic backgrounds and levels of intelligence. At the 
same time, there was a small degree of variation (students from different Navy 
backgrounds) that Stuart and Shamdasani called for in the groups. 

• Since it is preferable to have two or more analysts coding focus group results 
independently, this technique has proven to be suitable for a multi-person research 
team. 

• The rolling interview technique allowed us to learn as we went along and to benefit 
from conducting multiple groups. 

• As previously mentioned, a simple descriptive narrative, rather than technically 
detailed analysis of the focus groups conducted is the most appropriate method for 
analyzing our data. 



4. Protocol Analysis in our Research 

Twelve subjects who had participated in the focus group inteiwiews 
volunteered for the protocol analysis portion of the data collection. Tliis exercise started 
with each subject participating in a few short warm-up examples to get him/her 
accustomed to thinking aloud. Following these exercises, participants were handed 
written instructions, followed by a question-answer segment, during which clarification 
of their questions was provided. The exercises were conducted individually, with each 
subject working in a semi-private area and his/her thoughts, as they were verbalized, were 
tape recorded. The researchers monitored the sessions to operate the audio equipment and 
to prompt the subjects to verbalize their thoughts, when necessary. Each session lasted 
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from 30 to 75 minutes. The recordings were iransciibed verbatim, and then searches were 
conducted throughout the transcripts for key words, phrases, concepts, or ideas that dealt 
with issues relating to traceability. 

In view of the proven track record of protocol analysis in numerous diverse 
areas, it was decided this technique could also prove beneficial to our research. Some 
specific reasons for choosing this method include: 



• Protocol Analysis was expected to provide detailed information on problem-solving 
with traceability information. 

• A sufficient number of subjects who had prior exposure to concepts of traceability 
in domains other than computer based systems development and who had 
participated in a system.s development (as a part of the case study) were readily 
available. 

• The issues under contention, mentioned in Section C above, are minimized in our 
study since the safeguards discussed were implemented during the exercise. 

E. SUMMARY 

This chapter provided an overview of the two data- collection techniques employed, 
and why they were appropriate lor our re.search. 'I'he next chapter will provide a specific 
analysis of the data collected. 
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IV. ANALYSIS OF DATA 



A. INTRODUCTION 

In this chapter, we discuss results from the analysis of data collected during focus 
groups and protocol analysis. First, we review the context in which traceability 
information is likely to be used during systems development; i.e. from the perspectives 
of key stakeholders involved in the systems development process. This is followed by 
a discussion of major issues that need to be addressed in the development of a model of 
traceability, and the mechanisms to support the capture of and reasoning with this 
infoimation. Findings from relevant liteiatui'e aie included to elucidate the main issues. 

B. STAKEHOLDERS 

A number of stakeholders are involved in the systems development process, 
including project sponsors, project managers, analysts, designers, maintainers, and end 
users. The development of a model of traceability should be geared towards these various 
stakeholders in the systems development process. This section will address these key 
stakeholders and what their concerns/uses for traceability encompass. 

1. Project Sponsor 

A project sponsor is the individual or organization that provides funding for 
the system being developed. ("The project sponsor is mostly concerned about cost 
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overruns and a finished product.")' Besides assuring the sponsor that genuine 
requirements are met, traceability also provides a mechanism to verify that unnecessary 
("wouldn’t it be nice to have") features are curtailed. In so doing, the sponsor can avoid 
potential cost overruns, schedule slippages, etc. 

2. Project Manager 

The project manager is the supervisor who "plans, delegates, and controls 
progress to develop an acceptable system within the allotted time and budget" (Whitten, 
et. al., 1989). He/she is the key person held accountable for a project from start to finish, 
and needs to ensure that al! the requirements are met. In general, he should make sure 
the project is finished on time, within the given budget, and that ("the project/system does 
what it was intended to"). A project manager uses such techniques as tracking 
milestones, etc., to ensure his/her responsibilities are accomplished. ("The project 
manager needs traceability for ... tracking milestones and ... keeping tabs on projects.") 
According to Brown, "Traceability provides for ease in determining phase completion and 
product completeness" (Brown, 1987). Traceability will also help the project manager 
determine when all requirements have been fully satisfied. 

3. Systems Desigiier/Aiialyst 

"A (software) designer often needs to trace from requirements objects to the 
corresponding design objects or from source code to its corresponding design or 

'This is a direct quote from a subject participating in the protocol analysis exercise. 
Henceforth, all quotes from a subject, made either in a focus group or during the protocol 
analysis exercise, will be enclosed in parentheses and quotation marks, but no specific 
reference will be made. 
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requiremenls objects" (Nejmeh el al, 1989, p. 981). This use of traceability will help a 
systems designer determine if all requirements have been considered and specifications 
validated. Further, the designer needs to understand why design objects satisfy particular 
requirements. ("A systems designer wants traceability in order to go back to the logic.”) 

The systems designer is also involved following systems implementation. 
("To the systems designer, traceability is extremely important as far as implementation 
goes ... [because hej is going to have to accommodate any design changes and 
[determine] the relative impact within the organization. If they don’t have good 
traceability in the system, he [systems designer) may implement a change which ... may 
even cripple the system.") 

4. Maintainer 

Maintainers are the personnel who make repairs to the system, once it has 
been implemented, and updates it to keep up with changing requirements. Once a change 
is required, a maintainer needs to he able to trace that change back to the requiremenls 
that necessitated or triggered it, and to pinpoint which parts of design/implementation are 
affected by the change. ("The systems maintainer wants traceability for ... tracing to a 
piece of code, for updating, and h)r changeability.") 

5. End Users 

Different levels of end users will employ traceability in varying degrees. On 
one end of the spectrum is the casual end u.ser, "one who may use only a specific on-line 



28 



program on an occasional basis" and "may never become truly comfortable with the 
terminal or the program" (Whitten, et. al., 1989). An example of a casual end user is a 
data entry clerk. On the other end of the spectrum is the dedicated end user, "one who 
will spend considerable lime using specific on-line programs. This user is likely to 
become comfortable and familiar with the lerminal’s operation" (ibid, 1989). 

The casual end user may have little or no need for traceability. ("(Casual end 
users) arc pretty much just concenied about using the system and don’t really care or 
have any power over where it came from or why it is the way it is."; "I wouldn’t see 
where they would be interested in the traceability of the design and the functionality of 
the system.") 

Dedicated end users, however, have more applications for traceability. Some 
subjects noted; ("The more sophisticated end user needs traceability to manuals to see 
how to achieve the functionality specified in recjuiremenis documents^ and traceability 
to programs, via queries, to modify them to achieve functionalities."; "For the dedicated 
end user, traceability is beneficial for understanding reasoning ... and for 
troubleshooting."). As for the end user manager, ("He needs traceability for 
accountability,’ for attempting to improve on a prototype ... and to enhance 
documentation."). 



^Traceability to documents is discussed further in a later section. 
’Accountability is addressed in a later section. 
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C. MAJOR ISSUES 



Our studies brought out several issues that need to be carefully examined to 
facilitate the development of a model of and mechanisms to capture and use traceability 
information. Following are some key issues we discovered, both in focus groups and 
protocol analysis, while evaluating the data: 

1. Bidirectional Traceability 

Bidirectional traceability implies both forward and backward traceability. 
Forward traceability is provided if each requirement specifically references a design 
component. In other words, forward traceability allows one actually to see where 
requirements materialize in the finished system. 

In the context of software design, forward traceability ... is especially important 
when the software product enters the operations and maintenance phase. As code 
and design documents are modified, it is essential to be able to ascertain the 
complete set of requirements that may be affected by those modifications (Thayer 
and Dorfman, 1990). 

Backwards traceability is provided when a requirement is referenced by a 
design element. In the context of definition of requirements from source documents, 
"Backward traceability ... to previous stages of development depends upon each 
requirement explicitly referencing its source in previous documents" (Dorfman and 
Thayer, 1990). Here, bidirectional traceability indicates that a requirement derived from 
a former requirement has been considered, and that any new requirement can be traced 
back to a preceding one. 

Though one of the most critical uses of traceability is ensuring that a design 
element satisfies a requirement, the existence of such a link may not answer the question: 
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are Uie funciionalilies of the design elemeni required by requiremems? To help answer 
this question, links need to be bidirectional in order to allow requirements to be traced 
forward from requirements to systems components, and backward, from systems 
components to requirements. 

2. Criticality of Requirements 

A useful way of identifying critical lequirements is to relate them to the 
central "mission" of the system. Business processes or missions that generate 
requirements could be identilled, and requirements evaluated with respect to such 
processes, to arrive at a elassificalion. For example, traceability should address the issue 
of how the requirements are arrived at. This necessitates a mechanism to represent the 
elaboration and refinement of requirements, from the central mission or business 
processes that generate them. A good traceability scheme should recognize that all 
requirements are not equal in level of signillcance or criticality. Different levels of detail 
must be established in order to minimize the overhead involved in capturing and using 
traceability infomiation. It may be unneeessary or even undesirable, considering the 
overhead involved in maintaining traceability, to maintain linkages between every 
requirement and every output created during the systems design process related to it. 
Costs must be justified by the benefits. It is essential to identify critical requirements and 
maintain traceability IVoni those requirements to the various systems components. 

The need to relate mission criticality to a traceability scheme was considered 
important by many subjects in the focus groups; ("We just have to realize that it 
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[traceability] is not necessary lor mundane decisions."; "Traceability is great for the 
critical stuff."). 

3. Design Rationale 

Another important component of traceability is design rationale information. 
On the need for design rationale MacLean states: "To understand why a system design 
is the way it is, we also need to understand how it could be different and why die choices 
which were made are appropriate" (MacLean. et al. 1989). 

Traceability linkages to represent rationale would capture the why or reason 
for design decisions. De.sign rationale allows for reasoning about a system’s 
characteristics in the process of understanding and changing it. Design rationale is an 
important issue in change management, as it can facilitate change while not necessarily 
providing the mechanism for doing so. Tracking relationships among design objects, and 
understanding how and which of iho.sc objects is affected by change, is vital in the 
maintenance of the system. 

The focus group pariiciiiants were keenly aware of the need for design 
rationale as a component of traceability: ("The systents designer needs traceability in 
order to examine the logic behind the system."; "Traceability could be very useful for 
justifying why you did something tlie way you did it."; "Traceability would be good for 
deteiTuining what input and output are required."; "We have some artificiality built into 
the system— you can say this is how it’s supposed to be, but is it leally? You may need 
traceability to help you adjust requirements."). 
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4. Project Tracking and Management 



Requirements traceability can be used veiy productively in project 
management and tracking. During the systems definition and subsequent phases, 
traceability is essential to ensure that all systems requirements have been met. 
Establishing all life cycle phases as complete can go a long way toward guaranteeing the 
ease of the verification and validation process. 

The project manager can use links such as stains, completion date, and 
authorization between various components of the system for scheduling, continuity, and 
security. Such information is indispensable in integrating project management into the 
systems development operation, and the efficient completion of project management ta.sks. 

Focus group participants were very interested in project tracking and 
management possibilities using traceability. ("Without traceability, if you’ve lost a 
linkage you spend much valuable time tracing back to the original requirement."; "If you 
don’t write down your thought processes and assumptions, and most people don’t, you 
can’t remember what you’ve been doing unless you have traceability."; "Humans don’t 
go back to the requirements enough."; "Traceability should be extremely helpful with 
tracking costs."; "The project manager needs traceability for tracking milestones."; 
"Traceability would be great for the project manager’s security concerns.") 

5. Accountability 

A major use of traceability is to provide accountability. Using traceability 
legitimately to communicate with the original designer of a system component, or to 
understand the capability of a system, is an example of such potential use. However, 
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caution must be used when employing traceability inrunnation to enforce accountability. 
The use of accountability information ns a means for perfonnance appraisal may be 
inappropriate. A parallel could be drawn to the use of information gathered duiing 
structured walkthroughs in systems development which should be strictly used for 
understanding and improving the current system and not for performance evaluation. 

Some accountability information that could be captured using traceability 
linkages include: design elements designed by, validated by, and modified by development 
personnel. The availability of such information will be indispensable in maintaining and 
revising a system. 

The focus group subjects perceived an urgent need for the accountability 
element tempered by constraints on its usage: ("Traceability needs to be something that 
humans can work with, not Just a whip held over people."; "Traceability should not be 
used to threaten people with."; "Accountability needs to be supplemented with good 
communication."). They were also mindful of the future: ("Accountability implies 

affoidability--we’re going to have less resources available In the future."; "I’m sure that 
I’m going to want to look back in the future and ask myself who made certain decisions 
or where decisions came from."). 

6. lliiinamvare 

Humanware form a critical component of any large-scale system. It just as 
important to capture how requirements relate to humanware as with other components of 
the system. This may involve tracing the responsibility for a requirement to a human. 



34 



A comprehensive mechanism for iraceability should link ihe human ware 
component of a system to the other components. Examples of such linkages include 
systems functionalities performed by humans. This information is necessary to ensure that 
the allocation of requirements is complete and conect. Focus group participants touched 
on the concept of humanware: ("We need traceability for human manageability."). 

7. Docuinents/Mamials 

Document traceability determines the existence of relationships between two 
document segments; it means that a particular document is in accord with a previous 
document, with which it has some type of relationship. Document traceability also 
ensures that all components in one document have a related component in another 
document. 

Consistency and completeness constraints apply within a document and across 
documents. Within a recpiimment specification, a requirement description may 
define inputs and outputs which relate to other requirements in the specification. 
Inconsistent references and incomplete specifications may occur and can be 
checked... (Horowitz and Williamson, 1986). 

Traceability linkages to documents include inierpreted by, defined by, and 
consistent with. Such linkages specify how to obtain a required peifonnance from a 
systems component. Our focus group subjects had considerable insight into some 

of the document traceability implications; ("Stakeholders aie interested in having 
traceability to be able to write quality manuals and data dictionaries."; "Traceability is 
good in that it de-emphasizes [unnecessary duplication in] documentation."; "If 
traceability is good, another contractor should be able to do documentation."; "Traceability 
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is nol a requiremcnl of docuineiuaiioii, but ii is highly desirable for documeiuation 

purposes.")- 

8. Dependencies 

Since complex sysiems are composed of inierdependeiU components, such 
dependencies should be represenied and muiniained. Often the inler-component 
dependencies are nol well undersiood and documented. 

Systems design is a complex activity involving interdependent decisions. In 
the absence of mechanisms to record such dependencies, over lime and with changing 
development teams, this information will be lost. Such dependencies may span diffeient 
systems components. A decision about software may be dependent on an earlier decision 
about hardware. As the system evolves over its life cycle, the hardware decision may be 
altered, leading to inconsistencies with the software that was based on the earlier 
hardware decision. Unless the dependencies are captured and maintained, such issues 
may go undetected, leading to .severe system integration problems. 

Another form of dependency is the fact that there may be several components 
needed to satisfy a re(|uiremeni. As the system evolves over its development life cycle, 
it is desirable to identify design or implementation elements that "partially satisfy" a given 
requirement. For instance, a hardware/sofiware combination is often necessary to satisfy 
a given requirement. When either the hardware or software component is developed, 
traceability information should mflect the fact that the partially satisfied requirements are 
fully satisfied by performance of necessary actions. 
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It is possible to identily a combination of design elements that satisfy a 
lequirement or are generated by it. An example of such a traceability scheme is the use 
of AND-OR graphs to represent traceability linkages. Such AND-OR graphs can be used 
to model a task in terms of a series of goals and subgoals. If requirements are treated as 
goals to be satisfied, the successive refinements can be treated as subgoals to be satisfied. 
The goals which can be satisfied only when all of their immediate subgoals, are satisfied 
are represented by AND nodes. When goals can be satisfied by any of the subgoals, they 
are represented by OR nodes. Liu and Horowitz (Liu and Horowitz, 1989) model the 
Work Breakdown Structure (WBS) of a software project as an AND-OR graph. This 
concept can be used in maintaining traceability linkages between various levels of outputs, 
when a logical combination of lower level outputs satisfies a liigher level goal or 
requirement. An AND-OR graph is depicted in Figure 1. 

Yet another form of dependency can be summed up: ("The data base design 
has a transitive dependency."). This dependency is identified when the ("data base design 
requires the data fiow diagram [which in turn] depends on the requirements. Tlierefore, 
requirements determine database design."). 

9. Horizontal and Vertical Traceability 

Vertical traceability refers to the "association of software (system) life cycle 
(SLC) objects of different types (typically created in different SLC processes). An 
example of a vertical traceability relationship would be between requirements statement 
and design statement" (Nejmeh et al, 1989, p. 981). ("Vertical traceability is easy because 
there’s a ’rule’ ... you explode a process and either you have to or you don’t."). 



37 




Figure 1. Example of AND-OR Graph 



Hoiizonial traceability rerers to the "association of SLC objects of the same 
type (typically created in the same SLC process). An example of this type of traceability 
includes parent/child relationships among decomposed data How diagrams and the 
’derived from’ relationship among requirement statements" (Nejmeh et al, 1989, p. 981). 
("When you’re moving horizontal is when you’re analyzing what process is inside what 
process."). Horizontal traceability equates to a ("subprocess transfeiring data to another 
subprocess like primitive levels have to talk to each other, etc."). 

Horizontal traceability also rerers to relationships between different views of the 
same level of (design) repre.senlation. For example, the relationships between the 
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behavioral view and t'unclional view ol'the system [Hoang, 91] could provide horizontal 
traceability. 

10. Automated Support for Traceability 

Automated support for traceability can be extremely valuable when systems 
are large and/or complex. "When performed manually, the tasks are time-consuming and 
error-prone; moreover, users’ abilities to analyze traceability data are limited by the sheer 
volume of data ..." (Ncjmch et al, 1989, p. 981). In such circumstances, "an automated 
software tool is an imperative, as the measuring process can become extremely onerous" 
(Shepperd and Ince, 1990). As stated by Thayer and Dorfman, "There have been many 
cases where it appeared, at the outset, that it would be an easy task to keep track of it 
[manually], but when the system design is complete, and the customer is trying to 
understand whether alt the test data really satisfies the original requirements they wrote, 
the automated traceability would be ’worth its weight in gold’" (Thayer and Dorfman, 
1990). 

The degree of automated support can vary widely, depending on the level of 
sophistication wairanted/desired. "The sim|)lest [form] is a list that is tabulated by the 
ID of the requirement" (ibid, 199t)). This list can be changed, as needed, to support the 
iteration process. The use of a llexible database program and other more intricate aids 
can be utilized for more complex automated support. 
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D. SUMMARY 



The issues reviewed above suygesl ihere me many aspects of traceability which 
need to be considered when contemplating a traceability model for real-time, complex 
systems. This chapter specifically indicates that different stakeholders will have different 
uses for traceability, and in varying degrees, llie next chapter provides a model of design 
rationale as an example of a complex traceability relationship to illustrate the concepts 
discussed here. 
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V. DESIGN RATIONALE AS AN EXAMPLE OF TRACEAUILITY 



A. INTRODUCTION 

In this chapier, we discuss a model for represeiiling and reasoning wilh design 
rationale as an example of a complex traceability scheme to illustrate and highlight some 
of the major issues discussed in the previous chapter. 

A conceptual model and mechanisms for the representation of and reasoning with 
process knowledge (i.e., design rationale) have been developed in earlier research as a 
part of the REMAP (Representation and Maintenance of Process Knowledge) project. The 
model and the mechanisms provided by REMAP for representing and reasoning with 
traceability information to support various stakeholders is discussed in detail elsewhere 
(Ramesh and Dhar, 1992). This design rationale model can be viewed as an instance of 
a traceability link between a reiiuiiement and a design element. The term "design 
element" denotes any pait of the system design or implementation (i.e., data How 
diagrams, specifications, pieces of hardware, humanware etc.). In this chapier, we discuss 
how such a model and reasoning mechanisms can be used in the context of the issues 
discussed in the previous chapter. 

B. ISSUES IN CAPTURE AND USE OF RATIONALE 
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1. Support Tor various stakeholders 

There are a variely of stakeholders involved in large software projects, each having 
a different set of goals and priorities. For each of the stakeholders, some useful support 
can be provided by I'ecording in some stmctured manner, the history of a design in the 
form of (design) rationale. 

2. Partially Satisfied Re(|uirenients 

The process of satisfying requirements may generate several issues that need to be 
resolved. Resolution of issues lead to one or more design components. Partially satisfied 
requirements may be identified with unresolved issues that 

relate to that requirement using structures like the AND-OR grajihs in REMAP. A similar 
structure can be used in linking design artifncts to lequiiements through design decisions. 

3. Criticality of Requirements 

Our model captures the elaboration and refinement of requirements. Critical 
’mission statements’ or core ’business process’ objectives can be the origin of such an 
elaboration and refinement. During this process, the ciiticality or importance of 
requirements can be a.scertained and monitored. The REMAP model can lepiesenl this 
information as an attribute of the links between mission statements/business processes and 
requirements or as attributes of requirements themselves. Then, the critical requirements 
can be monitored to ascertain whether all the issues related to them are resolved in a 
timely manner. 



42 



4. Qualitative and quantitative reasuning 

The streiigih or other characteristics of relationships can be either qualitative or 
quantitative. In REMAP, the contents of the primitives can be informal information (such 
as text). But the model has well defined semantics of relationships among its primitives, 
facilitating reasoning with this structure. For instance, the assumptions in a design 
situation can be given different degiees of belief (or validity), and these beliefs can be 
automatically propagated to beliefs in arguments, positions and so on. Further, the 
strengths can be either qualitative or quantitative. 

5. Change Maungeiiicnt 

In REMAP, changes to design rationale will automatically trigger changes in the 
belief status (or validity) of design solutions thereby suggesting redesign (Ramesh and 
Dhar, 1992). Since various components of the process knowledge that lead to the design 
solution are tightly related, changes to the constraint set resulting out of changed 
assumptions, decisions or requirements will initiate the synthesis of a new design solution 
and provide rich information to estimate the effort involved in redesign. 

6. Project Management 

REMAP provides facilities for representing and reasoning with temporal infomiation 
which can be useful for project management. Fur instance, a validity time can be assigned 
to issues which could be interpreted as the time frame during which that issue must be 
resolved. Then, this information can be used for generating reminders to the designers or 
managers to focus their attention on issues that may have to be resolved within a time 
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frame or used in rank ordering issues. Project planning and control can be facilitated by 
integrity constraints on its primitives. An example of such a constraint could state that no 
requirement can he elaborated or refined until all requirements with higher priority or 
earlier validity time are considered. 

7. Accountability 

The REMAP environment facilitates the automatic capture or the representation of 
accountability information associated with design rationale. 

8. Links to all system coinponeiils 

The REMAP model can be used to capture relationships between requirements and 
all system components, Including humanware, hardware, software etc. 

9. Automated Support 

REMAP provides automated support for different stakeholders including interactive 
querying and updating of the design rationale knowledge base, a client-server architecture 
for multi-user support, a textual as well as hypertext-like user interface to the knowledge 
base and a reason maintenance system for maintaining and reasoning with design 
rationale. 

10. Derived Links 

REMAP provides facilities for inferring knowledge based on deductive niles and 
facilitates the derivation of implicit links between requirements and design artifacts. For 
instance, a rule could state that if a design element is created by a decision, and the 
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decision resolves an issue and ihe issue was generaled by a requirement, then the design 
element traces to the requirement. 



C. SUMMARY 

Design rationale information supports a variety of stakeholders. A semantic model 
of design rationale such as the REMAP model illustrated here is essential for providing 
such support, facilitating reasoning with such knowledge. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 



A. INTRODUCTION 

In this chapter, based on the findings discussed in Chapter IV, an initial model of 
traceability is presented. Further, recommendations on methodologies to be used in a 
comprehensive study on traceability are presented. Specifically, the appropriateness of the 
two techniques used in this research are discussed. 

B. INITIAL MODEL 

The findings from the preliminary study suggest that a comprehensive model of 
traceability needs to be developed. Our approach to developing a model is to understand 
the traceability needs of various stakeholders in the systems development process. In order 
to fully support the stakeholders needs, our research suggests that a comprehensive model 
of traceability should capture semantic information (as does the REMAP model for design 
rationale) to allow for advanced reasoning with the traceability data. An initial attempt 
at a traceability model is shown in Figure 2 which demonstrates linkage, linkage types 
and ways to combine the link.s. 

In this figure, various link.s are shown between different stages of the development 
process (e.g, requirements and design). Recursive relationships are shown to denote both 
vertical (e.g., between high level and low level design) and horizontal (e.g., between 
different representations or views at the same level of design) linkages within a 
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development stage. Each link consists of at least one traceability type. At present our 
model consist of five types of links as denoted in Figure 2. These types were derived 
from the analysis of data explained in Chapter IV. An example follows: Requirement 1.3 
from document A is linked to function X (represented by dataflow diagram bubble 2.4). 
The links is of type satisfied which denotes that function X satisfies requirement 1.3. In 
some cases links may include more than one type. For example, the above link may be 
augmented by accountability information (e.g. the satisfied relationship was determined 
by Mr. Smith on Sept 10, 1992). 

In order to better support reasoning of the traceability infonnation, the model 
allows for several methods of combining the traceability links. The four methods for 
coinbining, shown in Figure 2, were derived based on the data collected. The need for 
a weighing scheme was noted by the discussion on criticality and the other three are 
described in detail in Chapter IV). An example of an and/or scheme was also presented 
in Figure 1. 

The authors believe that by using various types of links and methods for combining 
them, like the ones presented in this model, one can adequately capture the traceability 
infoimation and provide reasoning to satisfy the stakeholders previously mentioned needs. 

C. METHODOLOGIES FOR FUTURE RESEARCH 

In this I'esearch, two very powerful techniques were employed for data collection. 
Though the techniques are widely used in other disciplines, they are unconventional as 
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Figure 2. liiilial Traceiibilily Model 



empirical research lools in the domain of systems development. Comparison of the two 
primary data collection strategics provides some interesting insights into the research 
methodology appropriate for future work. Focus groups provided surprisingly interesting 
results. In this exploratory data collection method, the researcher’s biases do not constrain 
the participants. It should be noted that the moderators of the focus groups were non- 
experts in the domain, and therefore, the potential for bia.ses is minimal. In our study, 
many paiticipants related concepts of requirements traceability to their experiences in 
ship-building and aircraft maintenance which employ similar concepts. Focus groups 
conducted participants with real-life systems development experience is likely to provide 
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very valuable information, even if the participants are not very familiar with ament 
traceability tools and techniques, provided they have sufficient inteiest in the concept. As 
the participants aie not restricted by the researcher’s ideas and predispositions, this 
methodology will often provide new perspectives and approaches to the problem being 
explored. 

During the 1992 Complex Systems Engineering Synthesis and Assessment 
Technology Workshop held at the Naval Surface Warfaixi Center Dahlgren Division, a 
break-out session on Traceability wa.s held. Participants included Systems designers as 
well as CASE tool developers. The meeting had some of the characteristics of a focus 
group, though held in an informal selling. Several of ihe major issues that were identified 
in our study were raised by this informal focus groups, providing an infonnal validation 
of our findings. Further, as the group members had the characteristics similar to those of 
potential subjects in our comprehensive study, it is believed that focus groups as a 
methodology for data collection will be very valuable. 

Protocol analysis, on the other hand, is likely to provide very specific data on 
problem solving behavior. Very u.seful results can he obtained if the behavior is studied 
in a real-life problem solving situation. 'Ibis requirement severely constrains the use of 
protocol analysis in our future work. First, cunent methods of capturing and reasoning 
with traceability are inadequate to provide us an appropriate real-life problem solving 
situation. Second, the protocol analysis method is extremely expensive in terms of 
demands on the subjects and the researchers. Therefore, the use of this methodology 
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should be resuicted lo a very small mimbei of subjects in a relatively well-defined area 
of problems solving (e.g., traceability for accountability). 

D. CONCLUSIONS 

Our research provides very valuable insights into the development of a 
comprehensive model of traceability. Continued msearch needs to be done to refine and 
validate the model. The link types need to be further defined and the use of different 
traceability types in system development activities needs to be explored. The methods for 
combining links needs to be examined further. Further, automated methods for capture 
and analysis of links that involve various methods of combining them would be veiy 
useful. Over the next year the research intends to finalize the model and prioritize the 
types and combinations in terms of their importance in supporting various stakeholders. 
In future years each link type and combination method will be further investigated to 
enhance the model. 
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Appendix A. CASE STUDY DESCRIPTION 



The project provided an opportunity to learn about systems analysis and design by 
performing the analysis and design of a system based on a case study, Tlie case study had 
been developed after a detailed domain analysis. The domain analysis used data from a 
real-life large scale systems development effort. 

REQUIREMENTS 

The participants were recjuired to produce several outputs at various stages. 
PHASE 1: PROJECT PROPOSAL 

In this phase, the participants identified the applicatioji to be studied and provide 
the motivation for their project. 

A typical project propo.sal including the following; 

1. Name of the Project 

2. Brief description of the project 

2.1 Background 

2.2 Management 

2.3 Data Proce.ssing at the Organization 

2.4 Concerns over Information Systems issues that motivate the project 
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3. Brief background of each leain member and proposed division of (asks for ihe 
project (with Justification). 

4. Preliminary investigation to determine information requirements 

4.1 Interview imports (of key people involved) 

4.2 Summary of findings (This is typically a verbal description of the key 
elements found in all the interview conducted and reported in 1.1) 

4.3 Formalized Requirements Statements based on preliminary investigation 
(These were to be considered similar to those produced during Govt, 
systems development efforts) 

PHASE 2: SYSTEMS ANALYSIS 

Typically, in this phase, data flow diagrams, entity-relationship diagrams and data 
dictionaries are developed for the system. The basic tasks performed by the participants 
included; 

1. Data Flow Analysis 

This analysis consists of developing data flow diagrams, which describe the 
processes and tlie data dictionary, which defines system elements. Various 
CASE tools were used to develop the components. 

2. Entity-Relationship Model 

This analysis involves the u.se of the entity-relationship model to document 
the systems’s data independently of how the data will be used. 
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PHASE 3: SYSTEMS DESIGN 



The participanls designed and iinplenienled the systems using a relational database 
system. The database design phase included development of the logical schema and 
normalization. Hie project also included the development of a man-machine interface of 
the system being designed. Application generators were used to create input and output 
layouts. 
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CASE STUDY PROJECT TOPICS 



The parlicipanls were chose one of ihe four following subsyslems of a customer order 
processing system for a utility company. 

Subsystem 1: Telephone Answer Center System 

This subsystem deals with the oiierations of the telephone answer center. The 
primary users of the system are likely to be the telephone answer center operators 
and supei'visors. 

Subsystem 2; Field Station System 

This subsystem deals with the operations of the field offices for handling customer 
requests (except for appliance repair and maintenance). Tlie primary users of this 
system are likely to be the field supervisors and field technicians. 

Subsystem 3: Appliances Repair and Maintenances System 

This subsystem is a specialized field station system for handling appliance repairs 
and maintenance. I he primary users of this system are field supervisors (appliance) 
and field technicians. 
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Subsystem 4: Uilling System 



This subsystem deals with periodic compulation of bills to be provided to customers 
for the services provided by the utility company. The primai 7 user of this system 
is the billing department. 
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